{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "5a1d28f3",
   "metadata": {},
   "source": [
    "# Running Tune experiments with Nevergrad\n",
    "\n",
    "<a id=\"try-anyscale-quickstart-ray-tune-nevergrad_example\" href=\"https://console.anyscale.com/register/ha?render_flow=ray&utm_source=ray_docs&utm_medium=docs&utm_campaign=ray-tune-nevergrad_example\">\n",
    "    <img src=\"../../_static/img/run-on-anyscale.svg\" alt=\"try-anyscale-quickstart\">\n",
    "</a>\n",
    "<br></br>\n",
    "\n",
    "In this tutorial we introduce Nevergrad, while running a simple Ray Tune experiment. Tune’s Search Algorithms integrate with Nevergrad and, as a result, allow you to seamlessly scale up a Nevergrad optimization process - without sacrificing performance.\n",
    "\n",
    "Nevergrad provides gradient/derivative-free optimization able to handle noise over the objective landscape, including evolutionary, bandit, and Bayesian optimization algorithms. Nevergrad internally supports search spaces which are continuous, discrete or a mixture of thereof. It also provides a library of functions on which to test the optimization algorithms and compare with other benchmarks.\n",
    "\n",
    "In this example we minimize a simple objective to briefly demonstrate the usage of Nevergrad with Ray Tune via `NevergradSearch`. It's useful to keep in mind that despite the emphasis on machine learning experiments, Ray Tune optimizes any implicit or explicit objective. Here we assume `nevergrad==0.4.3.post7` library is installed. To learn more, please refer to [Nevergrad website](https://github.com/facebookresearch/nevergrad).\n",
    "\n",
    "Necessary requirements:\n",
    "- `pip install ray[tune] nevergrad==0.4.3.post7`\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "5ab54f85",
   "metadata": {
    "tags": [
     "remove-cell"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting nevergrad==0.4.3.post7\n",
      "  Using cached nevergrad-0.4.3.post7-py3-none-any.whl (400 kB)\n",
      "Collecting cma>=2.6.0\n",
      "  Downloading cma-3.2.2-py2.py3-none-any.whl (249 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m249.1/249.1 kB\u001b[0m \u001b[31m3.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0ma \u001b[36m0:00:01\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: bayesian-optimization>=1.2.0 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from nevergrad==0.4.3.post7) (1.2.0)\n",
      "Requirement already satisfied: numpy>=1.15.0 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from nevergrad==0.4.3.post7) (1.21.6)\n",
      "Requirement already satisfied: typing-extensions>=3.6.6 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from nevergrad==0.4.3.post7) (4.1.1)\n",
      "Requirement already satisfied: scipy>=0.14.0 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from bayesian-optimization>=1.2.0->nevergrad==0.4.3.post7) (1.4.1)\n",
      "Requirement already satisfied: scikit-learn>=0.18.0 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from bayesian-optimization>=1.2.0->nevergrad==0.4.3.post7) (0.24.2)\n",
      "Requirement already satisfied: threadpoolctl>=2.0.0 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from scikit-learn>=0.18.0->bayesian-optimization>=1.2.0->nevergrad==0.4.3.post7) (3.0.0)\n",
      "Requirement already satisfied: joblib>=0.11 in /Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages (from scikit-learn>=0.18.0->bayesian-optimization>=1.2.0->nevergrad==0.4.3.post7) (1.1.0)\n",
      "Installing collected packages: cma, nevergrad\n",
      "Successfully installed cma-3.2.2 nevergrad-0.4.3.post7\n",
      "\u001b[33mWARNING: There was an error checking the latest version of pip.\u001b[0m\u001b[33m\n",
      "\u001b[0m"
     ]
    }
   ],
   "source": [
    "# !pip install ray[tune]\n",
    "!pip install nevergrad==0.4.3.post7 "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "66cb8206",
   "metadata": {},
   "source": [
    "Click below to see all the imports we need for this example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "1f6d7a31",
   "metadata": {
    "tags": [
     "hide-input"
    ]
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/kai/.pyenv/versions/3.7.7/lib/python3.7/site-packages/nevergrad/optimization/differentialevolution.py:107: InefficientSettingsWarning: DE algorithms are inefficient with budget < 60\n",
      "  \"DE algorithms are inefficient with budget < 60\", base.errors.InefficientSettingsWarning\n"
     ]
    }
   ],
   "source": [
    "import time\n",
    "\n",
    "import ray\n",
    "import nevergrad as ng\n",
    "from ray import tune\n",
    "from ray.tune.search import ConcurrencyLimiter\n",
    "from ray.tune.search.nevergrad import NevergradSearch"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "41f2c881",
   "metadata": {},
   "source": [
    "Let's start by defining a simple evaluation function.\n",
    "We artificially sleep for a bit (`0.1` seconds) to simulate a long-running ML experiment.\n",
    "This setup assumes that we're running multiple `step`s of an experiment and try to tune two hyperparameters,\n",
    "namely `width` and `height`, and `activation`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "271bd5c5",
   "metadata": {},
   "outputs": [],
   "source": [
    "def evaluate(step, width, height, activation):\n",
    "    time.sleep(0.1)\n",
    "    activation_boost = 10 if activation==\"relu\" else 1\n",
    "    return (0.1 + width * step / 100) ** (-1) + height * 0.1 + activation_boost"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "f060ea83",
   "metadata": {},
   "source": [
    "Next, our `objective` function takes a Tune `config`, evaluates the `score` of your experiment in a training loop,\n",
    "and uses `tune.report` to report the `score` back to Tune."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "c71fc423",
   "metadata": {},
   "outputs": [],
   "source": [
    "def objective(config):\n",
    "    for step in range(config[\"steps\"]):\n",
    "        score = evaluate(step, config[\"width\"], config[\"height\"], config[\"activation\"])\n",
    "        tune.report({\"iterations\": step, \"mean_loss\": score})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "619263ee",
   "metadata": {
    "lines_to_next_cell": 0,
    "tags": [
     "remove-cell"
    ]
   },
   "outputs": [],
   "source": [
    "ray.init(configure_logging=False)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "5b7a4b94",
   "metadata": {},
   "source": [
    "Now we construct the hyperparameter search space using `ConfigSpace`"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "e2405373",
   "metadata": {},
   "source": [
    "Next we define the search algorithm built from `NevergradSearch`, constrained  to a maximum of `4` concurrent trials with a `ConcurrencyLimiter`. Here we use `ng.optimizers.OnePlusOne`, a simple evolutionary algorithm."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "f099b674",
   "metadata": {},
   "outputs": [],
   "source": [
    "algo = NevergradSearch(\n",
    "    optimizer=ng.optimizers.OnePlusOne,\n",
    ")\n",
    "algo = tune.search.ConcurrencyLimiter(algo, max_concurrent=4)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "4bddc4e5",
   "metadata": {},
   "source": [
    "The number of samples is the number of hyperparameter combinations that will be tried out. This Tune run is set to `1000` samples.\n",
    "(you can decrease this if it takes too long on your machine)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "adb807bc",
   "metadata": {},
   "outputs": [],
   "source": [
    "num_samples = 1000"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "191c7f89",
   "metadata": {
    "tags": [
     "remove-cell"
    ]
   },
   "outputs": [],
   "source": [
    "# If 1000 samples take too long, you can reduce this number.\n",
    "# We override this number here for our smoke tests.\n",
    "num_samples = 10"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "a3956381",
   "metadata": {},
   "source": [
    "Finally, all that's left is to define a search space."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "8829ffc5",
   "metadata": {},
   "outputs": [],
   "source": [
    "search_config = {\n",
    "    \"steps\": 100,\n",
    "    \"width\": tune.uniform(0, 20),\n",
    "    \"height\": tune.uniform(-100, 100),\n",
    "    \"activation\": tune.choice([\"relu, tanh\"])\n",
    "}"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "0b9d051f",
   "metadata": {},
   "source": [
    "Finally, we run the experiment to `\"min\"`imize the \"mean_loss\" of the `objective` by searching `search_space` via `algo`, `num_samples` times. This previous sentence is fully characterizes the search problem we aim to solve. With this in mind, observe how efficient it is to execute `tuner.fit()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "769f4368",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:ray.tune.trainable.function_trainable:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "== Status ==<br>Current time: 2022-07-22 15:24:44 (running for 00:00:43.69)<br>Memory usage on this node: 10.4/16.0 GiB<br>Using FIFO scheduling algorithm.<br>Resources requested: 0/16 CPUs, 0/0 GPUs, 0.0/4.61 GiB heap, 0.0/2.0 GiB objects<br>Current best trial: 004f499a with mean_loss=-7.595329711238255 and parameters={'steps': 100, 'width': 2.1174116156230918, 'height': -90.50653873694615, 'activation': 'relu, tanh'}<br>Result logdir: /Users/kai/ray_results/objective_2022-07-22_15-23-59<br>Number of trials: 10/10 (10 TERMINATED)<br><table>\n",
       "<thead>\n",
       "<tr><th>Trial name        </th><th>status    </th><th>loc            </th><th>activation  </th><th style=\"text-align: right;\">   height</th><th style=\"text-align: right;\">    width</th><th style=\"text-align: right;\">    loss</th><th style=\"text-align: right;\">  iter</th><th style=\"text-align: right;\">  total time (s)</th><th style=\"text-align: right;\">  iterations</th><th style=\"text-align: right;\">  neg_mean_loss</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "<tr><td>objective_ee2ca136</td><td>TERMINATED</td><td>127.0.0.1:46434</td><td>relu, tanh  </td><td style=\"text-align: right;\">  0      </td><td style=\"text-align: right;\">10       </td><td style=\"text-align: right;\"> 1.1    </td><td style=\"text-align: right;\">   100</td><td style=\"text-align: right;\">         10.942 </td><td style=\"text-align: right;\">          99</td><td style=\"text-align: right;\">       -1.1    </td></tr>\n",
       "<tr><td>objective_efe1626e</td><td>TERMINATED</td><td>127.0.0.1:46441</td><td>relu, tanh  </td><td style=\"text-align: right;\">-31.0013 </td><td style=\"text-align: right;\"> 9.28761 </td><td style=\"text-align: right;\">-1.99254</td><td style=\"text-align: right;\">   100</td><td style=\"text-align: right;\">         11.5354</td><td style=\"text-align: right;\">          99</td><td style=\"text-align: right;\">        1.99254</td></tr>\n",
       "<tr><td>objective_efe34e4e</td><td>TERMINATED</td><td>127.0.0.1:46442</td><td>relu, tanh  </td><td style=\"text-align: right;\">  5.21403</td><td style=\"text-align: right;\"> 9.48974 </td><td style=\"text-align: right;\"> 1.62672</td><td style=\"text-align: right;\">   100</td><td style=\"text-align: right;\">         11.6606</td><td style=\"text-align: right;\">          99</td><td style=\"text-align: right;\">       -1.62672</td></tr>\n",
       "<tr><td>objective_efe55c2a</td><td>TERMINATED</td><td>127.0.0.1:46443</td><td>relu, tanh  </td><td style=\"text-align: right;\">-20.8721 </td><td style=\"text-align: right;\">11.3958  </td><td style=\"text-align: right;\">-0.99935</td><td style=\"text-align: right;\">   100</td><td style=\"text-align: right;\">         11.6083</td><td style=\"text-align: right;\">          99</td><td style=\"text-align: right;\">        0.99935</td></tr>\n",
       "<tr><td>objective_f6688086</td><td>TERMINATED</td><td>127.0.0.1:46467</td><td>relu, tanh  </td><td style=\"text-align: right;\"> 57.2829 </td><td style=\"text-align: right;\">17.7296  </td><td style=\"text-align: right;\"> 6.78493</td><td style=\"text-align: right;\">   100</td><td style=\"text-align: right;\">         10.716 </td><td style=\"text-align: right;\">          99</td><td style=\"text-align: right;\">       -6.78493</td></tr>\n",
       "<tr><td>objective_f85ed926</td><td>TERMINATED</td><td>127.0.0.1:46478</td><td>relu, tanh  </td><td style=\"text-align: right;\"> 40.5543 </td><td style=\"text-align: right;\">19.0813  </td><td style=\"text-align: right;\"> 5.10809</td><td style=\"text-align: right;\">   100</td><td style=\"text-align: right;\">         10.7158</td><td style=\"text-align: right;\">          99</td><td style=\"text-align: right;\">       -5.10809</td></tr>\n",
       "<tr><td>objective_f86ee276</td><td>TERMINATED</td><td>127.0.0.1:46481</td><td>relu, tanh  </td><td style=\"text-align: right;\"> 93.8686 </td><td style=\"text-align: right;\"> 1.60757 </td><td style=\"text-align: right;\">10.9781 </td><td style=\"text-align: right;\">   100</td><td style=\"text-align: right;\">         10.7415</td><td style=\"text-align: right;\">          99</td><td style=\"text-align: right;\">      -10.9781 </td></tr>\n",
       "<tr><td>objective_f880a02e</td><td>TERMINATED</td><td>127.0.0.1:46484</td><td>relu, tanh  </td><td style=\"text-align: right;\">-80.5769 </td><td style=\"text-align: right;\"> 5.84852 </td><td style=\"text-align: right;\">-6.88791</td><td style=\"text-align: right;\">   100</td><td style=\"text-align: right;\">         10.7335</td><td style=\"text-align: right;\">          99</td><td style=\"text-align: right;\">        6.88791</td></tr>\n",
       "<tr><td>objective_fe4e7a44</td><td>TERMINATED</td><td>127.0.0.1:46499</td><td>relu, tanh  </td><td style=\"text-align: right;\">  9.62911</td><td style=\"text-align: right;\"> 0.622909</td><td style=\"text-align: right;\"> 3.35823</td><td style=\"text-align: right;\">   100</td><td style=\"text-align: right;\">         13.1428</td><td style=\"text-align: right;\">          99</td><td style=\"text-align: right;\">       -3.35823</td></tr>\n",
       "<tr><td>objective_004f499a</td><td>TERMINATED</td><td>127.0.0.1:46504</td><td>relu, tanh  </td><td style=\"text-align: right;\">-90.5065 </td><td style=\"text-align: right;\"> 2.11741 </td><td style=\"text-align: right;\">-7.59533</td><td style=\"text-align: right;\">   100</td><td style=\"text-align: right;\">         10.7688</td><td style=\"text-align: right;\">          99</td><td style=\"text-align: right;\">        7.59533</td></tr>\n",
       "</tbody>\n",
       "</table><br><br>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:ray._private.runtime_env.plugin_schema_manager:Loading the default runtime env schemas: ['/Users/kai/coding/ray/python/ray/_private/runtime_env/../../runtime_env/schemas/working_dir_schema.json', '/Users/kai/coding/ray/python/ray/_private/runtime_env/../../runtime_env/schemas/pip_schema.json'].\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Result for objective_ee2ca136:\n",
      "  date: 2022-07-22_15-24-03\n",
      "  done: false\n",
      "  experiment_id: c0ad5ddb78cc4cc88e8195f5bde34e20\n",
      "  hostname: Kais-MacBook-Pro.local\n",
      "  iterations: 0\n",
      "  iterations_since_restore: 1\n",
      "  mean_loss: 11.0\n",
      "  neg_mean_loss: -11.0\n",
      "  node_ip: 127.0.0.1\n",
      "  pid: 46434\n",
      "  time_since_restore: 0.10390329360961914\n",
      "  time_this_iter_s: 0.10390329360961914\n",
      "  time_total_s: 0.10390329360961914\n",
      "  timestamp: 1658499843\n",
      "  timesteps_since_restore: 0\n",
      "  training_iteration: 1\n",
      "  trial_id: ee2ca136\n",
      "  warmup_time: 0.003058910369873047\n",
      "  \n",
      "Result for objective_efe1626e:\n",
      "  date: 2022-07-22_15-24-06\n",
      "  done: false\n",
      "  experiment_id: c6d33a9a30c040c0929b657f1d3e1557\n",
      "  hostname: Kais-MacBook-Pro.local\n",
      "  iterations: 0\n",
      "  iterations_since_restore: 1\n",
      "  mean_loss: 7.899868000941147\n",
      "  neg_mean_loss: -7.899868000941147\n",
      "  node_ip: 127.0.0.1\n",
      "  pid: 46441\n",
      "  time_since_restore: 0.10202908515930176\n",
      "  time_this_iter_s: 0.10202908515930176\n",
      "  time_total_s: 0.10202908515930176\n",
      "  timestamp: 1658499846\n",
      "  timesteps_since_restore: 0\n",
      "  training_iteration: 1\n",
      "  trial_id: efe1626e\n",
      "  warmup_time: 0.003210783004760742\n",
      "  \n",
      "Result for objective_efe55c2a:\n",
      "  date: 2022-07-22_15-24-06\n",
      "  done: false\n",
      "  experiment_id: 903e9605ba894aa0bc55297229b8a77b\n",
      "  hostname: Kais-MacBook-Pro.local\n",
      "  iterations: 0\n",
      "  iterations_since_restore: 1\n",
      "  mean_loss: 8.91279068144465\n",
      "  neg_mean_loss: -8.91279068144465\n",
      "  node_ip: 127.0.0.1\n",
      "  pid: 46443\n",
      "  time_since_restore: 0.1029670238494873\n",
      "  time_this_iter_s: 0.1029670238494873\n",
      "  time_total_s: 0.1029670238494873\n",
      "  timestamp: 1658499846\n",
      "  timesteps_since_restore: 0\n",
      "  training_iteration: 1\n",
      "  trial_id: efe55c2a\n",
      "  warmup_time: 0.0031630992889404297\n",
      "  \n",
      "Result for objective_efe34e4e:\n",
      "  date: 2022-07-22_15-24-06\n",
      "  done: false\n",
      "  experiment_id: 1efbb9c1becb436e8f304b61e9edd61b\n",
      "  hostname: Kais-MacBook-Pro.local\n",
      "  iterations: 0\n",
      "  iterations_since_restore: 1\n",
      "  mean_loss: 11.521403250268252\n",
      "  neg_mean_loss: -11.521403250268252\n",
      "  node_ip: 127.0.0.1\n",
      "  pid: 46442\n",
      "  time_since_restore: 0.10443997383117676\n",
      "  time_this_iter_s: 0.10443997383117676\n",
      "  time_total_s: 0.10443997383117676\n",
      "  timestamp: 1658499846\n",
      "  timesteps_since_restore: 0\n",
      "  training_iteration: 1\n",
      "  trial_id: efe34e4e\n",
      "  warmup_time: 0.002650022506713867\n",
      "  \n",
      "Result for objective_ee2ca136:\n",
      "  date: 2022-07-22_15-24-08\n",
      "  done: false\n",
      "  experiment_id: c0ad5ddb78cc4cc88e8195f5bde34e20\n",
      "  hostname: Kais-MacBook-Pro.local\n",
      "  iterations: 45\n",
      "  iterations_since_restore: 46\n",
      "  mean_loss: 1.2173913043478262\n",
      "  neg_mean_loss: -1.2173913043478262\n",
      "  node_ip: 127.0.0.1\n",
      "  pid: 46434\n",
      "  time_since_restore: 5.145823955535889\n",
      "  time_this_iter_s: 0.10791492462158203\n",
      "  time_total_s: 5.145823955535889\n",
      "  timestamp: 1658499848\n",
      "  timesteps_since_restore: 0\n",
      "  training_iteration: 46\n",
      "  trial_id: ee2ca136\n",
      "  warmup_time: 0.003058910369873047\n",
      "  \n",
      "Result for objective_efe1626e:\n",
      "  date: 2022-07-22_15-24-11\n",
      "  done: false\n",
      "  experiment_id: c6d33a9a30c040c0929b657f1d3e1557\n",
      "  hostname: Kais-MacBook-Pro.local\n",
      "  iterations: 47\n",
      "  iterations_since_restore: 48\n",
      "  mean_loss: -1.8761766831351419\n",
      "  neg_mean_loss: 1.8761766831351419\n",
      "  node_ip: 127.0.0.1\n",
      "  pid: 46441\n",
      "  time_since_restore: 5.138395071029663\n",
      "  time_this_iter_s: 0.10561490058898926\n",
      "  time_total_s: 5.138395071029663\n",
      "  timestamp: 1658499851\n",
      "  timesteps_since_restore: 0\n",
      "  training_iteration: 48\n",
      "  trial_id: efe1626e\n",
      "  warmup_time: 0.003210783004760742\n",
      "  \n",
      "Result for objective_efe55c2a:\n",
      "  date: 2022-07-22_15-24-11\n",
      "  done: false\n",
      "  experiment_id: 903e9605ba894aa0bc55297229b8a77b\n",
      "  hostname: Kais-MacBook-Pro.local\n",
      "  iterations: 47\n",
      "  iterations_since_restore: 48\n",
      "  mean_loss: -0.9039251143764186\n",
      "  neg_mean_loss: 0.9039251143764186\n",
      "  node_ip: 127.0.0.1\n",
      "  pid: 46443\n",
      "  time_since_restore: 5.145689249038696\n",
      "  time_this_iter_s: 0.10677504539489746\n",
      "  time_total_s: 5.145689249038696\n",
      "  timestamp: 1658499851\n",
      "  timesteps_since_restore: 0\n",
      "  training_iteration: 48\n",
      "  trial_id: efe55c2a\n",
      "  warmup_time: 0.0031630992889404297\n",
      "  \n",
      "Result for objective_efe34e4e:\n",
      "  date: 2022-07-22_15-24-11\n",
      "  done: false\n",
      "  experiment_id: 1efbb9c1becb436e8f304b61e9edd61b\n",
      "  hostname: Kais-MacBook-Pro.local\n",
      "  iterations: 47\n",
      "  iterations_since_restore: 48\n",
      "  mean_loss: 1.7406930109818743\n",
      "  neg_mean_loss: -1.7406930109818743\n",
      "  node_ip: 127.0.0.1\n",
      "  pid: 46442\n",
      "  time_since_restore: 5.151263952255249\n",
      "  time_this_iter_s: 0.10529589653015137\n",
      "  time_total_s: 5.151263952255249\n",
      "  timestamp: 1658499851\n",
      "  timesteps_since_restore: 0\n",
      "  training_iteration: 48\n",
      "  trial_id: efe34e4e\n",
      "  warmup_time: 0.002650022506713867\n",
      "  \n",
      "Result for objective_ee2ca136:\n",
      "  date: 2022-07-22_15-24-13\n",
      "  done: false\n",
      "  experiment_id: c0ad5ddb78cc4cc88e8195f5bde34e20\n",
      "  hostname: Kais-MacBook-Pro.local\n",
      "  iterations: 92\n",
      "  iterations_since_restore: 93\n",
      "  mean_loss: 1.10752688172043\n",
      "  neg_mean_loss: -1.10752688172043\n",
      "  node_ip: 127.0.0.1\n",
      "  pid: 46434\n",
      "  time_since_restore: 10.185918092727661\n",
      "  time_this_iter_s: 0.10853385925292969\n",
      "  time_total_s: 10.185918092727661\n",
      "  timestamp: 1658499853\n",
      "  timesteps_since_restore: 0\n",
      "  training_iteration: 93\n",
      "  trial_id: ee2ca136\n",
      "  warmup_time: 0.003058910369873047\n",
      "  \n",
      "Result for objective_ee2ca136:\n",
      "  date: 2022-07-22_15-24-14\n",
      "  done: true\n",
      "  experiment_id: c0ad5ddb78cc4cc88e8195f5bde34e20\n",
      "  experiment_tag: 1_activation=relu_tanh,height=0.0000,steps=100,width=10.0000\n",
      "  hostname: Kais-MacBook-Pro.local\n",
      "  iterations: 99\n",
      "  iterations_since_restore: 100\n",
      "  mean_loss: 1.1\n",
      "  neg_mean_loss: -1.1\n",
      "  node_ip: 127.0.0.1\n",
      "  pid: 46434\n",
      "  time_since_restore: 10.941971063613892\n",
      "  time_this_iter_s: 0.10557413101196289\n",
      "  time_total_s: 10.941971063613892\n",
      "  timestamp: 1658499854\n",
      "  timesteps_since_restore: 0\n",
      "  training_iteration: 100\n",
      "  trial_id: ee2ca136\n",
      "  warmup_time: 0.003058910369873047\n",
      "  \n",
      "Result for objective_efe34e4e:\n",
      "  date: 2022-07-22_15-24-16\n",
      "  done: false\n",
      "  experiment_id: 1efbb9c1becb436e8f304b61e9edd61b\n",
      "  hostname: Kais-MacBook-Pro.local\n",
      "  iterations: 90\n",
      "  iterations_since_restore: 91\n",
      "  mean_loss: 1.63713376556457\n",
      "  neg_mean_loss: -1.63713376556457\n",
      "  node_ip: 127.0.0.1\n",
      "  pid: 46442\n",
      "  time_since_restore: 9.770322799682617\n",
      "  time_this_iter_s: 0.1040806770324707\n",
      "  time_total_s: 9.770322799682617\n",
      "  timestamp: 1658499856\n",
      "  timesteps_since_restore: 0\n",
      "  training_iteration: 91\n",
      "  trial_id: efe34e4e\n",
      "  warmup_time: 0.002650022506713867\n",
      "  \n",
      "Result for objective_efe1626e:\n",
      "  date: 2022-07-22_15-24-16\n",
      "  done: false\n",
      "  experiment_id: c6d33a9a30c040c0929b657f1d3e1557\n",
      "  hostname: Kais-MacBook-Pro.local\n",
      "  iterations: 92\n",
      "  iterations_since_restore: 93\n",
      "  mean_loss: -1.9844528559710652\n",
      "  neg_mean_loss: 1.9844528559710652\n",
      "  node_ip: 127.0.0.1\n",
      "  pid: 46441\n",
      "  time_since_restore: 9.955276012420654\n",
      "  time_this_iter_s: 0.10721087455749512\n",
      "  time_total_s: 9.955276012420654\n",
      "  timestamp: 1658499856\n",
      "  timesteps_since_restore: 0\n",
      "  training_iteration: 93\n",
      "  trial_id: efe1626e\n",
      "  warmup_time: 0.003210783004760742\n",
      "  \n",
      "Result for objective_efe55c2a:\n",
      "  date: 2022-07-22_15-24-16\n",
      "  done: false\n",
      "  experiment_id: 903e9605ba894aa0bc55297229b8a77b\n",
      "  hostname: Kais-MacBook-Pro.local\n",
      "  iterations: 91\n",
      "  iterations_since_restore: 92\n",
      "  mean_loss: -0.991699632507838\n",
      "  neg_mean_loss: 0.991699632507838\n",
      "  node_ip: 127.0.0.1\n",
      "  pid: 46443\n",
      "  time_since_restore: 9.83866286277771\n",
      "  time_this_iter_s: 0.10593676567077637\n",
      "  time_total_s: 9.83866286277771\n",
      "  timestamp: 1658499856\n",
      "  timesteps_since_restore: 0\n",
      "  training_iteration: 92\n",
      "  trial_id: efe55c2a\n",
      "  warmup_time: 0.0031630992889404297\n",
      "  \n",
      "Result for objective_f6688086:\n",
      "  date: 2022-07-22_15-24-17\n",
      "  done: false\n",
      "  experiment_id: aa090556819b41c5b1e93d761dc9311a\n",
      "  hostname: Kais-MacBook-Pro.local\n",
      "  iterations: 0\n",
      "  iterations_since_restore: 1\n",
      "  mean_loss: 16.728285222315503\n",
      "  neg_mean_loss: -16.728285222315503\n",
      "  node_ip: 127.0.0.1\n",
      "  pid: 46467\n",
      "  time_since_restore: 0.10288572311401367\n",
      "  time_this_iter_s: 0.10288572311401367\n",
      "  time_total_s: 0.10288572311401367\n",
      "  timestamp: 1658499857\n",
      "  timesteps_since_restore: 0\n",
      "  training_iteration: 1\n",
      "  trial_id: f6688086\n",
      "  warmup_time: 0.002875089645385742\n",
      "  \n",
      "Result for objective_efe1626e:\n",
      "  date: 2022-07-22_15-24-17\n",
      "  done: true\n",
      "  experiment_id: c6d33a9a30c040c0929b657f1d3e1557\n",
      "  experiment_tag: 2_activation=relu_tanh,height=-31.0013,steps=100,width=9.2876\n",
      "  hostname: Kais-MacBook-Pro.local\n",
      "  iterations: 99\n",
      "  iterations_since_restore: 100\n",
      "  mean_loss: -1.9925441896649678\n",
      "  neg_mean_loss: 1.9925441896649678\n",
      "  node_ip: 127.0.0.1\n",
      "  pid: 46441\n",
      "  time_since_restore: 11.535439252853394\n",
      "  time_this_iter_s: 0.10611414909362793\n",
      "  time_total_s: 11.535439252853394\n",
      "  timestamp: 1658499857\n",
      "  timesteps_since_restore: 0\n",
      "  training_iteration: 100\n",
      "  trial_id: efe1626e\n",
      "  warmup_time: 0.003210783004760742\n",
      "  \n",
      "Result for objective_efe55c2a:\n",
      "  date: 2022-07-22_15-24-17\n",
      "  done: true\n",
      "  experiment_id: 903e9605ba894aa0bc55297229b8a77b\n",
      "  experiment_tag: 4_activation=relu_tanh,height=-20.8721,steps=100,width=11.3958\n",
      "  hostname: Kais-MacBook-Pro.local\n",
      "  iterations: 99\n",
      "  iterations_since_restore: 100\n",
      "  mean_loss: -0.9993497773424045\n",
      "  neg_mean_loss: 0.9993497773424045\n",
      "  node_ip: 127.0.0.1\n",
      "  pid: 46443\n",
      "  time_since_restore: 11.608299970626831\n",
      "  time_this_iter_s: 0.10889291763305664\n",
      "  time_total_s: 11.608299970626831\n",
      "  timestamp: 1658499857\n",
      "  timesteps_since_restore: 0\n",
      "  training_iteration: 100\n",
      "  trial_id: efe55c2a\n",
      "  warmup_time: 0.0031630992889404297\n",
      "  \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Result for objective_efe34e4e:\n",
      "  date: 2022-07-22_15-24-18\n",
      "  done: true\n",
      "  experiment_id: 1efbb9c1becb436e8f304b61e9edd61b\n",
      "  experiment_tag: 3_activation=relu_tanh,height=5.2140,steps=100,width=9.4897\n",
      "  hostname: Kais-MacBook-Pro.local\n",
      "  iterations: 99\n",
      "  iterations_since_restore: 100\n",
      "  mean_loss: 1.6267236167220034\n",
      "  neg_mean_loss: -1.6267236167220034\n",
      "  node_ip: 127.0.0.1\n",
      "  pid: 46442\n",
      "  time_since_restore: 11.660571098327637\n",
      "  time_this_iter_s: 0.1082301139831543\n",
      "  time_total_s: 11.660571098327637\n",
      "  timestamp: 1658499858\n",
      "  timesteps_since_restore: 0\n",
      "  training_iteration: 100\n",
      "  trial_id: efe34e4e\n",
      "  warmup_time: 0.002650022506713867\n",
      "  \n",
      "Result for objective_f85ed926:\n",
      "  date: 2022-07-22_15-24-20\n",
      "  done: false\n",
      "  experiment_id: 402684968ef24377ae7787c9af50d3ae\n",
      "  hostname: Kais-MacBook-Pro.local\n",
      "  iterations: 0\n",
      "  iterations_since_restore: 1\n",
      "  mean_loss: 15.055429995999226\n",
      "  neg_mean_loss: -15.055429995999226\n",
      "  node_ip: 127.0.0.1\n",
      "  pid: 46478\n",
      "  time_since_restore: 0.10394287109375\n",
      "  time_this_iter_s: 0.10394287109375\n",
      "  time_total_s: 0.10394287109375\n",
      "  timestamp: 1658499860\n",
      "  timesteps_since_restore: 0\n",
      "  training_iteration: 1\n",
      "  trial_id: f85ed926\n",
      "  warmup_time: 0.0031621456146240234\n",
      "  \n",
      "Result for objective_f86ee276:\n",
      "  date: 2022-07-22_15-24-20\n",
      "  done: false\n",
      "  experiment_id: cbdc4d94c3d64e59aa53e4465349dbe1\n",
      "  hostname: Kais-MacBook-Pro.local\n",
      "  iterations: 0\n",
      "  iterations_since_restore: 1\n",
      "  mean_loss: 20.386860838650087\n",
      "  neg_mean_loss: -20.386860838650087\n",
      "  node_ip: 127.0.0.1\n",
      "  pid: 46481\n",
      "  time_since_restore: 0.10312676429748535\n",
      "  time_this_iter_s: 0.10312676429748535\n",
      "  time_total_s: 0.10312676429748535\n",
      "  timestamp: 1658499860\n",
      "  timesteps_since_restore: 0\n",
      "  training_iteration: 1\n",
      "  trial_id: f86ee276\n",
      "  warmup_time: 0.002810955047607422\n",
      "  \n",
      "Result for objective_f880a02e:\n",
      "  date: 2022-07-22_15-24-20\n",
      "  done: false\n",
      "  experiment_id: fb5af0209b2b45eda67ab40960a48a99\n",
      "  hostname: Kais-MacBook-Pro.local\n",
      "  iterations: 0\n",
      "  iterations_since_restore: 1\n",
      "  mean_loss: 2.94231225430282\n",
      "  neg_mean_loss: -2.94231225430282\n",
      "  node_ip: 127.0.0.1\n",
      "  pid: 46484\n",
      "  time_since_restore: 0.10325503349304199\n",
      "  time_this_iter_s: 0.10325503349304199\n",
      "  time_total_s: 0.10325503349304199\n",
      "  timestamp: 1658499860\n",
      "  timesteps_since_restore: 0\n",
      "  training_iteration: 1\n",
      "  trial_id: f880a02e\n",
      "  warmup_time: 0.0027141571044921875\n",
      "  \n",
      "Result for objective_f6688086:\n",
      "  date: 2022-07-22_15-24-22\n",
      "  done: false\n",
      "  experiment_id: aa090556819b41c5b1e93d761dc9311a\n",
      "  hostname: Kais-MacBook-Pro.local\n",
      "  iterations: 47\n",
      "  iterations_since_restore: 48\n",
      "  mean_loss: 6.846867893893624\n",
      "  neg_mean_loss: -6.846867893893624\n",
      "  node_ip: 127.0.0.1\n",
      "  pid: 46467\n",
      "  time_since_restore: 5.122231721878052\n",
      "  time_this_iter_s: 0.10641169548034668\n",
      "  time_total_s: 5.122231721878052\n",
      "  timestamp: 1658499862\n",
      "  timesteps_since_restore: 0\n",
      "  training_iteration: 48\n",
      "  trial_id: f6688086\n",
      "  warmup_time: 0.002875089645385742\n",
      "  \n",
      "Result for objective_f85ed926:\n",
      "  date: 2022-07-22_15-24-25\n",
      "  done: false\n",
      "  experiment_id: 402684968ef24377ae7787c9af50d3ae\n",
      "  hostname: Kais-MacBook-Pro.local\n",
      "  iterations: 47\n",
      "  iterations_since_restore: 48\n",
      "  mean_loss: 5.1657053480548045\n",
      "  neg_mean_loss: -5.1657053480548045\n",
      "  node_ip: 127.0.0.1\n",
      "  pid: 46478\n",
      "  time_since_restore: 5.151860952377319\n",
      "  time_this_iter_s: 0.10704493522644043\n",
      "  time_total_s: 5.151860952377319\n",
      "  timestamp: 1658499865\n",
      "  timesteps_since_restore: 0\n",
      "  training_iteration: 48\n",
      "  trial_id: f85ed926\n",
      "  warmup_time: 0.0031621456146240234\n",
      "  \n",
      "Result for objective_f86ee276:\n",
      "  date: 2022-07-22_15-24-25\n",
      "  done: false\n",
      "  experiment_id: cbdc4d94c3d64e59aa53e4465349dbe1\n",
      "  hostname: Kais-MacBook-Pro.local\n",
      "  iterations: 47\n",
      "  iterations_since_restore: 48\n",
      "  mean_loss: 11.55568804118157\n",
      "  neg_mean_loss: -11.55568804118157\n",
      "  node_ip: 127.0.0.1\n",
      "  pid: 46481\n",
      "  time_since_restore: 5.140729665756226\n",
      "  time_this_iter_s: 0.1076810359954834\n",
      "  time_total_s: 5.140729665756226\n",
      "  timestamp: 1658499865\n",
      "  timesteps_since_restore: 0\n",
      "  training_iteration: 48\n",
      "  trial_id: f86ee276\n",
      "  warmup_time: 0.002810955047607422\n",
      "  \n",
      "Result for objective_f880a02e:\n",
      "  date: 2022-07-22_15-24-25\n",
      "  done: false\n",
      "  experiment_id: fb5af0209b2b45eda67ab40960a48a99\n",
      "  hostname: Kais-MacBook-Pro.local\n",
      "  iterations: 47\n",
      "  iterations_since_restore: 48\n",
      "  mean_loss: -6.706663109525936\n",
      "  neg_mean_loss: 6.706663109525936\n",
      "  node_ip: 127.0.0.1\n",
      "  pid: 46484\n",
      "  time_since_restore: 5.157264947891235\n",
      "  time_this_iter_s: 0.1107029914855957\n",
      "  time_total_s: 5.157264947891235\n",
      "  timestamp: 1658499865\n",
      "  timesteps_since_restore: 0\n",
      "  training_iteration: 48\n",
      "  trial_id: f880a02e\n",
      "  warmup_time: 0.0027141571044921875\n",
      "  \n",
      "Result for objective_f6688086:\n",
      "  date: 2022-07-22_15-24-27\n",
      "  done: false\n",
      "  experiment_id: aa090556819b41c5b1e93d761dc9311a\n",
      "  hostname: Kais-MacBook-Pro.local\n",
      "  iterations: 94\n",
      "  iterations_since_restore: 95\n",
      "  mean_loss: 6.787930201151392\n",
      "  neg_mean_loss: -6.787930201151392\n",
      "  node_ip: 127.0.0.1\n",
      "  pid: 46467\n",
      "  time_since_restore: 10.178911924362183\n",
      "  time_this_iter_s: 0.10588788986206055\n",
      "  time_total_s: 10.178911924362183\n",
      "  timestamp: 1658499867\n",
      "  timesteps_since_restore: 0\n",
      "  training_iteration: 95\n",
      "  trial_id: f6688086\n",
      "  warmup_time: 0.002875089645385742\n",
      "  \n",
      "Result for objective_f6688086:\n",
      "  date: 2022-07-22_15-24-27\n",
      "  done: true\n",
      "  experiment_id: aa090556819b41c5b1e93d761dc9311a\n",
      "  experiment_tag: 5_activation=relu_tanh,height=57.2829,steps=100,width=17.7296\n",
      "  hostname: Kais-MacBook-Pro.local\n",
      "  iterations: 99\n",
      "  iterations_since_restore: 100\n",
      "  mean_loss: 6.784934893475021\n",
      "  neg_mean_loss: -6.784934893475021\n",
      "  node_ip: 127.0.0.1\n",
      "  pid: 46467\n",
      "  time_since_restore: 10.71599006652832\n",
      "  time_this_iter_s: 0.10760498046875\n",
      "  time_total_s: 10.71599006652832\n",
      "  timestamp: 1658499867\n",
      "  timesteps_since_restore: 0\n",
      "  training_iteration: 100\n",
      "  trial_id: f6688086\n",
      "  warmup_time: 0.002875089645385742\n",
      "  \n",
      "Result for objective_fe4e7a44:\n",
      "  date: 2022-07-22_15-24-30\n",
      "  done: false\n",
      "  experiment_id: 95711fcb897a4f9fae3ebd811619fba9\n",
      "  hostname: Kais-MacBook-Pro.local\n",
      "  iterations: 0\n",
      "  iterations_since_restore: 1\n",
      "  mean_loss: 11.962911259228687\n",
      "  neg_mean_loss: -11.962911259228687\n",
      "  node_ip: 127.0.0.1\n",
      "  pid: 46499\n",
      "  time_since_restore: 0.10338783264160156\n",
      "  time_this_iter_s: 0.10338783264160156\n",
      "  time_total_s: 0.10338783264160156\n",
      "  timestamp: 1658499870\n",
      "  timesteps_since_restore: 0\n",
      "  training_iteration: 1\n",
      "  trial_id: fe4e7a44\n",
      "  warmup_time: 0.002811908721923828\n",
      "  \n",
      "Result for objective_f85ed926:\n",
      "  date: 2022-07-22_15-24-30\n",
      "  done: false\n",
      "  experiment_id: 402684968ef24377ae7787c9af50d3ae\n",
      "  hostname: Kais-MacBook-Pro.local\n",
      "  iterations: 94\n",
      "  iterations_since_restore: 95\n",
      "  mean_loss: 5.110873373928027\n",
      "  neg_mean_loss: -5.110873373928027\n",
      "  node_ip: 127.0.0.1\n",
      "  pid: 46478\n",
      "  time_since_restore: 10.178997039794922\n",
      "  time_this_iter_s: 0.10618400573730469\n",
      "  time_total_s: 10.178997039794922\n",
      "  timestamp: 1658499870\n",
      "  timesteps_since_restore: 0\n",
      "  training_iteration: 95\n",
      "  trial_id: f85ed926\n",
      "  warmup_time: 0.0031621456146240234\n",
      "  \n",
      "Result for objective_f86ee276:\n",
      "  date: 2022-07-22_15-24-30\n",
      "  done: false\n",
      "  experiment_id: cbdc4d94c3d64e59aa53e4465349dbe1\n",
      "  hostname: Kais-MacBook-Pro.local\n",
      "  iterations: 94\n",
      "  iterations_since_restore: 95\n",
      "  mean_loss: 11.007548256848812\n",
      "  neg_mean_loss: -11.007548256848812\n",
      "  node_ip: 127.0.0.1\n",
      "  pid: 46481\n",
      "  time_since_restore: 10.165759801864624\n",
      "  time_this_iter_s: 0.10771489143371582\n",
      "  time_total_s: 10.165759801864624\n",
      "  timestamp: 1658499870\n",
      "  timesteps_since_restore: 0\n",
      "  training_iteration: 95\n",
      "  trial_id: f86ee276\n",
      "  warmup_time: 0.002810955047607422\n",
      "  \n",
      "Result for objective_f880a02e:\n",
      "  date: 2022-07-22_15-24-30\n",
      "  done: false\n",
      "  experiment_id: fb5af0209b2b45eda67ab40960a48a99\n",
      "  hostname: Kais-MacBook-Pro.local\n",
      "  iterations: 94\n",
      "  iterations_since_restore: 95\n",
      "  mean_loss: -6.87903993853598\n",
      "  neg_mean_loss: 6.87903993853598\n",
      "  node_ip: 127.0.0.1\n",
      "  pid: 46484\n",
      "  time_since_restore: 10.188904047012329\n",
      "  time_this_iter_s: 0.10939908027648926\n",
      "  time_total_s: 10.188904047012329\n",
      "  timestamp: 1658499870\n",
      "  timesteps_since_restore: 0\n",
      "  training_iteration: 95\n",
      "  trial_id: f880a02e\n",
      "  warmup_time: 0.0027141571044921875\n",
      "  \n",
      "Result for objective_f85ed926:\n",
      "  date: 2022-07-22_15-24-31\n",
      "  done: true\n",
      "  experiment_id: 402684968ef24377ae7787c9af50d3ae\n",
      "  experiment_tag: 6_activation=relu_tanh,height=40.5543,steps=100,width=19.0813\n",
      "  hostname: Kais-MacBook-Pro.local\n",
      "  iterations: 99\n",
      "  iterations_since_restore: 100\n",
      "  mean_loss: 5.108087948450606\n",
      "  neg_mean_loss: -5.108087948450606\n",
      "  node_ip: 127.0.0.1\n",
      "  pid: 46478\n",
      "  time_since_restore: 10.71581220626831\n",
      "  time_this_iter_s: 0.10749602317810059\n",
      "  time_total_s: 10.71581220626831\n",
      "  timestamp: 1658499871\n",
      "  timesteps_since_restore: 0\n",
      "  training_iteration: 100\n",
      "  trial_id: f85ed926\n",
      "  warmup_time: 0.0031621456146240234\n",
      "  \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Result for objective_f86ee276:\n",
      "  date: 2022-07-22_15-24-31\n",
      "  done: true\n",
      "  experiment_id: cbdc4d94c3d64e59aa53e4465349dbe1\n",
      "  experiment_tag: 7_activation=relu_tanh,height=93.8686,steps=100,width=1.6076\n",
      "  hostname: Kais-MacBook-Pro.local\n",
      "  iterations: 99\n",
      "  iterations_since_restore: 100\n",
      "  mean_loss: 10.978053669828887\n",
      "  neg_mean_loss: -10.978053669828887\n",
      "  node_ip: 127.0.0.1\n",
      "  pid: 46481\n",
      "  time_since_restore: 10.741472005844116\n",
      "  time_this_iter_s: 0.1456291675567627\n",
      "  time_total_s: 10.741472005844116\n",
      "  timestamp: 1658499871\n",
      "  timesteps_since_restore: 0\n",
      "  training_iteration: 100\n",
      "  trial_id: f86ee276\n",
      "  warmup_time: 0.002810955047607422\n",
      "  \n",
      "Result for objective_f880a02e:\n",
      "  date: 2022-07-22_15-24-31\n",
      "  done: true\n",
      "  experiment_id: fb5af0209b2b45eda67ab40960a48a99\n",
      "  experiment_tag: 8_activation=relu_tanh,height=-80.5769,steps=100,width=5.8485\n",
      "  hostname: Kais-MacBook-Pro.local\n",
      "  iterations: 99\n",
      "  iterations_since_restore: 100\n",
      "  mean_loss: -6.887909370541976\n",
      "  neg_mean_loss: 6.887909370541976\n",
      "  node_ip: 127.0.0.1\n",
      "  pid: 46484\n",
      "  time_since_restore: 10.733515977859497\n",
      "  time_this_iter_s: 0.1060791015625\n",
      "  time_total_s: 10.733515977859497\n",
      "  timestamp: 1658499871\n",
      "  timesteps_since_restore: 0\n",
      "  training_iteration: 100\n",
      "  trial_id: f880a02e\n",
      "  warmup_time: 0.0027141571044921875\n",
      "  \n",
      "Result for objective_004f499a:\n",
      "  date: 2022-07-22_15-24-33\n",
      "  done: false\n",
      "  experiment_id: c3015ffd206444cd9c2eb1152aae5dd0\n",
      "  hostname: Kais-MacBook-Pro.local\n",
      "  iterations: 0\n",
      "  iterations_since_restore: 1\n",
      "  mean_loss: 1.9493461263053842\n",
      "  neg_mean_loss: -1.9493461263053842\n",
      "  node_ip: 127.0.0.1\n",
      "  pid: 46504\n",
      "  time_since_restore: 0.1044008731842041\n",
      "  time_this_iter_s: 0.1044008731842041\n",
      "  time_total_s: 0.1044008731842041\n",
      "  timestamp: 1658499873\n",
      "  timesteps_since_restore: 0\n",
      "  training_iteration: 1\n",
      "  trial_id: 004f499a\n",
      "  warmup_time: 0.0026237964630126953\n",
      "  \n",
      "Result for objective_fe4e7a44:\n",
      "  date: 2022-07-22_15-24-35\n",
      "  done: false\n",
      "  experiment_id: 95711fcb897a4f9fae3ebd811619fba9\n",
      "  hostname: Kais-MacBook-Pro.local\n",
      "  iterations: 25\n",
      "  iterations_since_restore: 26\n",
      "  mean_loss: 5.873327389463485\n",
      "  neg_mean_loss: -5.873327389463485\n",
      "  node_ip: 127.0.0.1\n",
      "  pid: 46499\n",
      "  time_since_restore: 5.199802875518799\n",
      "  time_this_iter_s: 0.1075749397277832\n",
      "  time_total_s: 5.199802875518799\n",
      "  timestamp: 1658499875\n",
      "  timesteps_since_restore: 0\n",
      "  training_iteration: 26\n",
      "  trial_id: fe4e7a44\n",
      "  warmup_time: 0.002811908721923828\n",
      "  \n",
      "Result for objective_004f499a:\n",
      "  date: 2022-07-22_15-24-38\n",
      "  done: false\n",
      "  experiment_id: c3015ffd206444cd9c2eb1152aae5dd0\n",
      "  hostname: Kais-MacBook-Pro.local\n",
      "  iterations: 47\n",
      "  iterations_since_restore: 48\n",
      "  mean_loss: -7.137564846035238\n",
      "  neg_mean_loss: 7.137564846035238\n",
      "  node_ip: 127.0.0.1\n",
      "  pid: 46504\n",
      "  time_since_restore: 5.15981912612915\n",
      "  time_this_iter_s: 0.10737729072570801\n",
      "  time_total_s: 5.15981912612915\n",
      "  timestamp: 1658499878\n",
      "  timesteps_since_restore: 0\n",
      "  training_iteration: 48\n",
      "  trial_id: 004f499a\n",
      "  warmup_time: 0.0026237964630126953\n",
      "  \n",
      "Result for objective_fe4e7a44:\n",
      "  date: 2022-07-22_15-24-40\n",
      "  done: false\n",
      "  experiment_id: 95711fcb897a4f9fae3ebd811619fba9\n",
      "  hostname: Kais-MacBook-Pro.local\n",
      "  iterations: 72\n",
      "  iterations_since_restore: 73\n",
      "  mean_loss: 3.7860835740628094\n",
      "  neg_mean_loss: -3.7860835740628094\n",
      "  node_ip: 127.0.0.1\n",
      "  pid: 46499\n",
      "  time_since_restore: 10.234857082366943\n",
      "  time_this_iter_s: 0.10819101333618164\n",
      "  time_total_s: 10.234857082366943\n",
      "  timestamp: 1658499880\n",
      "  timesteps_since_restore: 0\n",
      "  training_iteration: 73\n",
      "  trial_id: fe4e7a44\n",
      "  warmup_time: 0.002811908721923828\n",
      "  \n",
      "Result for objective_fe4e7a44:\n",
      "  date: 2022-07-22_15-24-43\n",
      "  done: true\n",
      "  experiment_id: 95711fcb897a4f9fae3ebd811619fba9\n",
      "  experiment_tag: 9_activation=relu_tanh,height=9.6291,steps=100,width=0.6229\n",
      "  hostname: Kais-MacBook-Pro.local\n",
      "  iterations: 99\n",
      "  iterations_since_restore: 100\n",
      "  mean_loss: 3.3582342398877607\n",
      "  neg_mean_loss: -3.3582342398877607\n",
      "  node_ip: 127.0.0.1\n",
      "  pid: 46499\n",
      "  time_since_restore: 13.142819166183472\n",
      "  time_this_iter_s: 0.10675215721130371\n",
      "  time_total_s: 13.142819166183472\n",
      "  timestamp: 1658499883\n",
      "  timesteps_since_restore: 0\n",
      "  training_iteration: 100\n",
      "  trial_id: fe4e7a44\n",
      "  warmup_time: 0.002811908721923828\n",
      "  \n",
      "Result for objective_004f499a:\n",
      "  date: 2022-07-22_15-24-43\n",
      "  done: false\n",
      "  experiment_id: c3015ffd206444cd9c2eb1152aae5dd0\n",
      "  hostname: Kais-MacBook-Pro.local\n",
      "  iterations: 94\n",
      "  iterations_since_restore: 95\n",
      "  mean_loss: -7.572268959036313\n",
      "  neg_mean_loss: 7.572268959036313\n",
      "  node_ip: 127.0.0.1\n",
      "  pid: 46504\n",
      "  time_since_restore: 10.228418827056885\n",
      "  time_this_iter_s: 0.10715484619140625\n",
      "  time_total_s: 10.228418827056885\n",
      "  timestamp: 1658499883\n",
      "  timesteps_since_restore: 0\n",
      "  training_iteration: 95\n",
      "  trial_id: 004f499a\n",
      "  warmup_time: 0.0026237964630126953\n",
      "  \n",
      "Result for objective_004f499a:\n",
      "  date: 2022-07-22_15-24-44\n",
      "  done: true\n",
      "  experiment_id: c3015ffd206444cd9c2eb1152aae5dd0\n",
      "  experiment_tag: 10_activation=relu_tanh,height=-90.5065,steps=100,width=2.1174\n",
      "  hostname: Kais-MacBook-Pro.local\n",
      "  iterations: 99\n",
      "  iterations_since_restore: 100\n",
      "  mean_loss: -7.595329711238255\n",
      "  neg_mean_loss: 7.595329711238255\n",
      "  node_ip: 127.0.0.1\n",
      "  pid: 46504\n",
      "  time_since_restore: 10.768790006637573\n",
      "  time_this_iter_s: 0.10774111747741699\n",
      "  time_total_s: 10.768790006637573\n",
      "  timestamp: 1658499884\n",
      "  timesteps_since_restore: 0\n",
      "  training_iteration: 100\n",
      "  trial_id: 004f499a\n",
      "  warmup_time: 0.0026237964630126953\n",
      "  \n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:ray.tune.tune:Total run time: 44.70 seconds (43.68 seconds for the tuning loop).\n"
     ]
    }
   ],
   "source": [
    "tuner = tune.Tuner(\n",
    "    objective,\n",
    "    tune_config=tune.TuneConfig(\n",
    "        metric=\"mean_loss\",\n",
    "        mode=\"min\",\n",
    "        search_alg=algo,\n",
    "        num_samples=num_samples,\n",
    "    ),\n",
    "    param_space=search_config,\n",
    ")\n",
    "results = tuner.fit()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "950003be",
   "metadata": {},
   "source": [
    "Here are the hyperparameters found to minimize the mean loss of the defined objective."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "0f021674",
   "metadata": {
    "lines_to_next_cell": 0
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best hyperparameters found were:  {'steps': 100, 'width': 2.1174116156230918, 'height': -90.50653873694615, 'activation': 'relu, tanh'}\n"
     ]
    }
   ],
   "source": [
    "print(\"Best hyperparameters found were: \", results.get_best_result().config)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "0d3824ae",
   "metadata": {},
   "source": [
    "## Optional: passing the (hyper)parameter space into the search algorithm\n",
    "\n",
    "We can also pass the search space into `NevergradSearch` using their designed format."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "89ae7455",
   "metadata": {},
   "outputs": [],
   "source": [
    "space = ng.p.Dict(\n",
    "    width=ng.p.Scalar(lower=0, upper=20),\n",
    "    height=ng.p.Scalar(lower=-100, upper=100),\n",
    "    activation=ng.p.Choice(choices=[\"relu\", \"tanh\"])\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "e52eeab1",
   "metadata": {},
   "outputs": [],
   "source": [
    "algo = NevergradSearch(\n",
    "    optimizer=ng.optimizers.OnePlusOne,\n",
    "    space=space,\n",
    "    metric=\"mean_loss\",\n",
    "    mode=\"min\"\n",
    ")\n",
    "algo = tune.search.ConcurrencyLimiter(algo, max_concurrent=4)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "f8926177",
   "metadata": {},
   "source": [
    "Again we run the experiment, this time with a less passed via the `config` and instead passed through `search_alg`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "64f39800",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "== Status ==<br>Current time: 2022-07-22 15:25:27 (running for 00:00:43.22)<br>Memory usage on this node: 10.8/16.0 GiB<br>Using FIFO scheduling algorithm.<br>Resources requested: 0/16 CPUs, 0/0 GPUs, 0.0/4.61 GiB heap, 0.0/2.0 GiB objects<br>Result logdir: /Users/kai/ray_results/objective_2022-07-22_15-24-44<br>Number of trials: 10/10 (10 TERMINATED)<br><table>\n",
       "<thead>\n",
       "<tr><th>Trial name        </th><th>status    </th><th>loc            </th><th>activation  </th><th style=\"text-align: right;\">   height</th><th style=\"text-align: right;\">   width</th><th style=\"text-align: right;\">     loss</th><th style=\"text-align: right;\">  iter</th><th style=\"text-align: right;\">  total time (s)</th><th style=\"text-align: right;\">  iterations</th><th style=\"text-align: right;\">  neg_mean_loss</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "<tr><td>objective_085274be</td><td>TERMINATED</td><td>127.0.0.1:46516</td><td>tanh        </td><td style=\"text-align: right;\">  0      </td><td style=\"text-align: right;\">10      </td><td style=\"text-align: right;\"> 1.1     </td><td style=\"text-align: right;\">   100</td><td style=\"text-align: right;\">         10.7324</td><td style=\"text-align: right;\">          99</td><td style=\"text-align: right;\">      -1.1     </td></tr>\n",
       "<tr><td>objective_09dee4f2</td><td>TERMINATED</td><td>127.0.0.1:46524</td><td>tanh        </td><td style=\"text-align: right;\">-44.4216 </td><td style=\"text-align: right;\">12.9653 </td><td style=\"text-align: right;\">-3.36485 </td><td style=\"text-align: right;\">   100</td><td style=\"text-align: right;\">         11.3476</td><td style=\"text-align: right;\">          99</td><td style=\"text-align: right;\">       3.36485 </td></tr>\n",
       "<tr><td>objective_09e0846a</td><td>TERMINATED</td><td>127.0.0.1:46525</td><td>relu        </td><td style=\"text-align: right;\">-38.0638 </td><td style=\"text-align: right;\">11.1574 </td><td style=\"text-align: right;\"> 6.28334 </td><td style=\"text-align: right;\">   100</td><td style=\"text-align: right;\">         11.3103</td><td style=\"text-align: right;\">          99</td><td style=\"text-align: right;\">      -6.28334 </td></tr>\n",
       "<tr><td>objective_09e21122</td><td>TERMINATED</td><td>127.0.0.1:46526</td><td>relu        </td><td style=\"text-align: right;\"> 41.2509 </td><td style=\"text-align: right;\"> 9.75585</td><td style=\"text-align: right;\">14.2276  </td><td style=\"text-align: right;\">   100</td><td style=\"text-align: right;\">         11.3512</td><td style=\"text-align: right;\">          99</td><td style=\"text-align: right;\">     -14.2276  </td></tr>\n",
       "<tr><td>objective_1045958e</td><td>TERMINATED</td><td>127.0.0.1:46544</td><td>relu        </td><td style=\"text-align: right;\">-73.2818 </td><td style=\"text-align: right;\"> 5.78832</td><td style=\"text-align: right;\"> 2.84334 </td><td style=\"text-align: right;\">   100</td><td style=\"text-align: right;\">         10.7372</td><td style=\"text-align: right;\">          99</td><td style=\"text-align: right;\">      -2.84334 </td></tr>\n",
       "<tr><td>objective_12309db2</td><td>TERMINATED</td><td>127.0.0.1:46549</td><td>relu        </td><td style=\"text-align: right;\">-94.9666 </td><td style=\"text-align: right;\">16.9764 </td><td style=\"text-align: right;\"> 0.562486</td><td style=\"text-align: right;\">   100</td><td style=\"text-align: right;\">         10.7329</td><td style=\"text-align: right;\">          99</td><td style=\"text-align: right;\">      -0.562486</td></tr>\n",
       "<tr><td>objective_12342770</td><td>TERMINATED</td><td>127.0.0.1:46550</td><td>tanh        </td><td style=\"text-align: right;\">-98.0775 </td><td style=\"text-align: right;\">17.2252 </td><td style=\"text-align: right;\">-8.74945 </td><td style=\"text-align: right;\">   100</td><td style=\"text-align: right;\">         10.7455</td><td style=\"text-align: right;\">          99</td><td style=\"text-align: right;\">       8.74945 </td></tr>\n",
       "<tr><td>objective_12374d7e</td><td>TERMINATED</td><td>127.0.0.1:46551</td><td>relu        </td><td style=\"text-align: right;\"> -1.60759</td><td style=\"text-align: right;\">18.0841 </td><td style=\"text-align: right;\"> 9.89479 </td><td style=\"text-align: right;\">   100</td><td style=\"text-align: right;\">         10.7348</td><td style=\"text-align: right;\">          99</td><td style=\"text-align: right;\">      -9.89479 </td></tr>\n",
       "<tr><td>objective_18344524</td><td>TERMINATED</td><td>127.0.0.1:46569</td><td>tanh        </td><td style=\"text-align: right;\">-41.1284 </td><td style=\"text-align: right;\">12.2952 </td><td style=\"text-align: right;\">-3.03135 </td><td style=\"text-align: right;\">   100</td><td style=\"text-align: right;\">         12.622 </td><td style=\"text-align: right;\">          99</td><td style=\"text-align: right;\">       3.03135 </td></tr>\n",
       "<tr><td>objective_1a1e29b8</td><td>TERMINATED</td><td>127.0.0.1:46576</td><td>tanh        </td><td style=\"text-align: right;\"> 64.0289 </td><td style=\"text-align: right;\">10.0482 </td><td style=\"text-align: right;\"> 7.50242 </td><td style=\"text-align: right;\">   100</td><td style=\"text-align: right;\">         10.8237</td><td style=\"text-align: right;\">          99</td><td style=\"text-align: right;\">      -7.50242 </td></tr>\n",
       "</tbody>\n",
       "</table><br><br>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Result for objective_085274be:\n",
      "  date: 2022-07-22_15-24-47\n",
      "  done: false\n",
      "  experiment_id: be25590fb790400ca28b548b8359a120\n",
      "  hostname: Kais-MacBook-Pro.local\n",
      "  iterations: 0\n",
      "  iterations_since_restore: 1\n",
      "  mean_loss: 11.0\n",
      "  neg_mean_loss: -11.0\n",
      "  node_ip: 127.0.0.1\n",
      "  pid: 46516\n",
      "  time_since_restore: 0.10224103927612305\n",
      "  time_this_iter_s: 0.10224103927612305\n",
      "  time_total_s: 0.10224103927612305\n",
      "  timestamp: 1658499887\n",
      "  timesteps_since_restore: 0\n",
      "  training_iteration: 1\n",
      "  trial_id: 085274be\n",
      "  warmup_time: 0.0029327869415283203\n",
      "  \n",
      "Result for objective_09dee4f2:\n",
      "  date: 2022-07-22_15-24-49\n",
      "  done: false\n",
      "  experiment_id: 9469ffc8df11476db00e329acd2ae8b6\n",
      "  hostname: Kais-MacBook-Pro.local\n",
      "  iterations: 0\n",
      "  iterations_since_restore: 1\n",
      "  mean_loss: 6.557843114006006\n",
      "  neg_mean_loss: -6.557843114006006\n",
      "  node_ip: 127.0.0.1\n",
      "  pid: 46524\n",
      "  time_since_restore: 0.10509586334228516\n",
      "  time_this_iter_s: 0.10509586334228516\n",
      "  time_total_s: 0.10509586334228516\n",
      "  timestamp: 1658499889\n",
      "  timesteps_since_restore: 0\n",
      "  training_iteration: 1\n",
      "  trial_id: 09dee4f2\n",
      "  warmup_time: 0.002938985824584961\n",
      "  \n",
      "Result for objective_09e21122:\n",
      "  date: 2022-07-22_15-24-49\n",
      "  done: false\n",
      "  experiment_id: 0a825198482b4bd4aded6c318dff0dcf\n",
      "  hostname: Kais-MacBook-Pro.local\n",
      "  iterations: 0\n",
      "  iterations_since_restore: 1\n",
      "  mean_loss: 24.125094234750687\n",
      "  neg_mean_loss: -24.125094234750687\n",
      "  node_ip: 127.0.0.1\n",
      "  pid: 46526\n",
      "  time_since_restore: 0.10476112365722656\n",
      "  time_this_iter_s: 0.10476112365722656\n",
      "  time_total_s: 0.10476112365722656\n",
      "  timestamp: 1658499889\n",
      "  timesteps_since_restore: 0\n",
      "  training_iteration: 1\n",
      "  trial_id: 09e21122\n",
      "  warmup_time: 0.003228902816772461\n",
      "  \n",
      "Result for objective_09e0846a:\n",
      "  date: 2022-07-22_15-24-49\n",
      "  done: false\n",
      "  experiment_id: 260410fc1fb24f78af507cd38a8361a9\n",
      "  hostname: Kais-MacBook-Pro.local\n",
      "  iterations: 0\n",
      "  iterations_since_restore: 1\n",
      "  mean_loss: 16.193619847512025\n",
      "  neg_mean_loss: -16.193619847512025\n",
      "  node_ip: 127.0.0.1\n",
      "  pid: 46525\n",
      "  time_since_restore: 0.10422873497009277\n",
      "  time_this_iter_s: 0.10422873497009277\n",
      "  time_total_s: 0.10422873497009277\n",
      "  timestamp: 1658499889\n",
      "  timesteps_since_restore: 0\n",
      "  training_iteration: 1\n",
      "  trial_id: 09e0846a\n",
      "  warmup_time: 0.002719879150390625\n",
      "  \n",
      "Result for objective_085274be:\n",
      "  date: 2022-07-22_15-24-52\n",
      "  done: false\n",
      "  experiment_id: be25590fb790400ca28b548b8359a120\n",
      "  hostname: Kais-MacBook-Pro.local\n",
      "  iterations: 47\n",
      "  iterations_since_restore: 48\n",
      "  mean_loss: 1.2083333333333333\n",
      "  neg_mean_loss: -1.2083333333333333\n",
      "  node_ip: 127.0.0.1\n",
      "  pid: 46516\n",
      "  time_since_restore: 5.132040977478027\n",
      "  time_this_iter_s: 0.1084139347076416\n",
      "  time_total_s: 5.132040977478027\n",
      "  timestamp: 1658499892\n",
      "  timesteps_since_restore: 0\n",
      "  training_iteration: 48\n",
      "  trial_id: 085274be\n",
      "  warmup_time: 0.0029327869415283203\n",
      "  \n",
      "Result for objective_09dee4f2:\n",
      "  date: 2022-07-22_15-24-54\n",
      "  done: false\n",
      "  experiment_id: 9469ffc8df11476db00e329acd2ae8b6\n",
      "  hostname: Kais-MacBook-Pro.local\n",
      "  iterations: 47\n",
      "  iterations_since_restore: 48\n",
      "  mean_loss: -3.280701838145472\n",
      "  neg_mean_loss: 3.280701838145472\n",
      "  node_ip: 127.0.0.1\n",
      "  pid: 46524\n",
      "  time_since_restore: 5.134023189544678\n",
      "  time_this_iter_s: 0.10428118705749512\n",
      "  time_total_s: 5.134023189544678\n",
      "  timestamp: 1658499894\n",
      "  timesteps_since_restore: 0\n",
      "  training_iteration: 48\n",
      "  trial_id: 09dee4f2\n",
      "  warmup_time: 0.002938985824584961\n",
      "  \n",
      "Result for objective_09e21122:\n",
      "  date: 2022-07-22_15-24-54\n",
      "  done: false\n",
      "  experiment_id: 0a825198482b4bd4aded6c318dff0dcf\n",
      "  hostname: Kais-MacBook-Pro.local\n",
      "  iterations: 47\n",
      "  iterations_since_restore: 48\n",
      "  mean_loss: 14.33852997141215\n",
      "  neg_mean_loss: -14.33852997141215\n",
      "  node_ip: 127.0.0.1\n",
      "  pid: 46526\n",
      "  time_since_restore: 5.139041185379028\n",
      "  time_this_iter_s: 0.10721492767333984\n",
      "  time_total_s: 5.139041185379028\n",
      "  timestamp: 1658499894\n",
      "  timesteps_since_restore: 0\n",
      "  training_iteration: 48\n",
      "  trial_id: 09e21122\n",
      "  warmup_time: 0.003228902816772461\n",
      "  \n",
      "Result for objective_09e0846a:\n",
      "  date: 2022-07-22_15-24-54\n",
      "  done: false\n",
      "  experiment_id: 260410fc1fb24f78af507cd38a8361a9\n",
      "  hostname: Kais-MacBook-Pro.local\n",
      "  iterations: 47\n",
      "  iterations_since_restore: 48\n",
      "  mean_loss: 6.3807469650477024\n",
      "  neg_mean_loss: -6.3807469650477024\n",
      "  node_ip: 127.0.0.1\n",
      "  pid: 46525\n",
      "  time_since_restore: 5.149268865585327\n",
      "  time_this_iter_s: 0.10705304145812988\n",
      "  time_total_s: 5.149268865585327\n",
      "  timestamp: 1658499894\n",
      "  timesteps_since_restore: 0\n",
      "  training_iteration: 48\n",
      "  trial_id: 09e0846a\n",
      "  warmup_time: 0.002719879150390625\n",
      "  \n",
      "Result for objective_085274be:\n",
      "  date: 2022-07-22_15-24-57\n",
      "  done: false\n",
      "  experiment_id: be25590fb790400ca28b548b8359a120\n",
      "  hostname: Kais-MacBook-Pro.local\n",
      "  iterations: 94\n",
      "  iterations_since_restore: 95\n",
      "  mean_loss: 1.1052631578947367\n",
      "  neg_mean_loss: -1.1052631578947367\n",
      "  node_ip: 127.0.0.1\n",
      "  pid: 46516\n",
      "  time_since_restore: 10.194181203842163\n",
      "  time_this_iter_s: 0.10754203796386719\n",
      "  time_total_s: 10.194181203842163\n",
      "  timestamp: 1658499897\n",
      "  timesteps_since_restore: 0\n",
      "  training_iteration: 95\n",
      "  trial_id: 085274be\n",
      "  warmup_time: 0.0029327869415283203\n",
      "  \n",
      "Result for objective_085274be:\n",
      "  date: 2022-07-22_15-24-57\n",
      "  done: true\n",
      "  experiment_id: be25590fb790400ca28b548b8359a120\n",
      "  experiment_tag: 1_activation=tanh,height=0.0000,steps=100,width=10.0000\n",
      "  hostname: Kais-MacBook-Pro.local\n",
      "  iterations: 99\n",
      "  iterations_since_restore: 100\n",
      "  mean_loss: 1.1\n",
      "  neg_mean_loss: -1.1\n",
      "  node_ip: 127.0.0.1\n",
      "  pid: 46516\n",
      "  time_since_restore: 10.732417106628418\n",
      "  time_this_iter_s: 0.10759997367858887\n",
      "  time_total_s: 10.732417106628418\n",
      "  timestamp: 1658499897\n",
      "  timesteps_since_restore: 0\n",
      "  training_iteration: 100\n",
      "  trial_id: 085274be\n",
      "  warmup_time: 0.0029327869415283203\n",
      "  \n",
      "Result for objective_09dee4f2:\n",
      "  date: 2022-07-22_15-24-59\n",
      "  done: false\n",
      "  experiment_id: 9469ffc8df11476db00e329acd2ae8b6\n",
      "  hostname: Kais-MacBook-Pro.local\n",
      "  iterations: 93\n",
      "  iterations_since_restore: 94\n",
      "  mean_loss: -3.3599044605358728\n",
      "  neg_mean_loss: 3.3599044605358728\n",
      "  node_ip: 127.0.0.1\n",
      "  pid: 46524\n",
      "  time_since_restore: 10.055138111114502\n",
      "  time_this_iter_s: 0.10467100143432617\n",
      "  time_total_s: 10.055138111114502\n",
      "  timestamp: 1658499899\n",
      "  timesteps_since_restore: 0\n",
      "  training_iteration: 94\n",
      "  trial_id: 09dee4f2\n",
      "  warmup_time: 0.002938985824584961\n",
      "  \n",
      "Result for objective_09e21122:\n",
      "  date: 2022-07-22_15-24-59\n",
      "  done: false\n",
      "  experiment_id: 0a825198482b4bd4aded6c318dff0dcf\n",
      "  hostname: Kais-MacBook-Pro.local\n",
      "  iterations: 93\n",
      "  iterations_since_restore: 94\n",
      "  mean_loss: 14.23411049568083\n",
      "  neg_mean_loss: -14.23411049568083\n",
      "  node_ip: 127.0.0.1\n",
      "  pid: 46526\n",
      "  time_since_restore: 10.066343307495117\n",
      "  time_this_iter_s: 0.1061861515045166\n",
      "  time_total_s: 10.066343307495117\n",
      "  timestamp: 1658499899\n",
      "  timesteps_since_restore: 0\n",
      "  training_iteration: 94\n",
      "  trial_id: 09e21122\n",
      "  warmup_time: 0.003228902816772461\n",
      "  \n",
      "Result for objective_09e0846a:\n",
      "  date: 2022-07-22_15-24-59\n",
      "  done: false\n",
      "  experiment_id: 260410fc1fb24f78af507cd38a8361a9\n",
      "  hostname: Kais-MacBook-Pro.local\n",
      "  iterations: 93\n",
      "  iterations_since_restore: 94\n",
      "  mean_loss: 6.2890729561522765\n",
      "  neg_mean_loss: -6.2890729561522765\n",
      "  node_ip: 127.0.0.1\n",
      "  pid: 46525\n",
      "  time_since_restore: 10.100499868392944\n",
      "  time_this_iter_s: 0.10625505447387695\n",
      "  time_total_s: 10.100499868392944\n",
      "  timestamp: 1658499899\n",
      "  timesteps_since_restore: 0\n",
      "  training_iteration: 94\n",
      "  trial_id: 09e0846a\n",
      "  warmup_time: 0.002719879150390625\n",
      "  \n",
      "Result for objective_1045958e:\n",
      "  date: 2022-07-22_15-25-00\n",
      "  done: false\n",
      "  experiment_id: 3579bfc2b346424b833f82f23d459807\n",
      "  hostname: Kais-MacBook-Pro.local\n",
      "  iterations: 0\n",
      "  iterations_since_restore: 1\n",
      "  mean_loss: 12.671822501738296\n",
      "  neg_mean_loss: -12.671822501738296\n",
      "  node_ip: 127.0.0.1\n",
      "  pid: 46544\n",
      "  time_since_restore: 0.10491013526916504\n",
      "  time_this_iter_s: 0.10491013526916504\n",
      "  time_total_s: 0.10491013526916504\n",
      "  timestamp: 1658499900\n",
      "  timesteps_since_restore: 0\n",
      "  training_iteration: 1\n",
      "  trial_id: 1045958e\n",
      "  warmup_time: 0.002847909927368164\n",
      "  \n",
      "Result for objective_09dee4f2:\n",
      "  date: 2022-07-22_15-25-01\n",
      "  done: true\n",
      "  experiment_id: 9469ffc8df11476db00e329acd2ae8b6\n",
      "  experiment_tag: 2_activation=tanh,height=-44.4216,steps=100,width=12.9653\n",
      "  hostname: Kais-MacBook-Pro.local\n",
      "  iterations: 99\n",
      "  iterations_since_restore: 100\n",
      "  mean_loss: -3.3648509190285747\n",
      "  neg_mean_loss: 3.3648509190285747\n",
      "  node_ip: 127.0.0.1\n",
      "  pid: 46524\n",
      "  time_since_restore: 11.347625017166138\n",
      "  time_this_iter_s: 0.10793185234069824\n",
      "  time_total_s: 11.347625017166138\n",
      "  timestamp: 1658499901\n",
      "  timesteps_since_restore: 0\n",
      "  training_iteration: 100\n",
      "  trial_id: 09dee4f2\n",
      "  warmup_time: 0.002938985824584961\n",
      "  \n",
      "Result for objective_09e0846a:\n",
      "  date: 2022-07-22_15-25-01\n",
      "  done: true\n",
      "  experiment_id: 260410fc1fb24f78af507cd38a8361a9\n",
      "  experiment_tag: 3_activation=relu,height=-38.0638,steps=100,width=11.1574\n",
      "  hostname: Kais-MacBook-Pro.local\n",
      "  iterations: 99\n",
      "  iterations_since_restore: 100\n",
      "  mean_loss: 6.28333982260153\n",
      "  neg_mean_loss: -6.28333982260153\n",
      "  node_ip: 127.0.0.1\n",
      "  pid: 46525\n",
      "  time_since_restore: 11.310342788696289\n",
      "  time_this_iter_s: 0.10942316055297852\n",
      "  time_total_s: 11.310342788696289\n",
      "  timestamp: 1658499901\n",
      "  timesteps_since_restore: 0\n",
      "  training_iteration: 100\n",
      "  trial_id: 09e0846a\n",
      "  warmup_time: 0.002719879150390625\n",
      "  \n",
      "Result for objective_09e21122:\n",
      "  date: 2022-07-22_15-25-01\n",
      "  done: true\n",
      "  experiment_id: 0a825198482b4bd4aded6c318dff0dcf\n",
      "  experiment_tag: 4_activation=relu,height=41.2509,steps=100,width=9.7559\n",
      "  hostname: Kais-MacBook-Pro.local\n",
      "  iterations: 99\n",
      "  iterations_since_restore: 100\n",
      "  mean_loss: 14.22757115653867\n",
      "  neg_mean_loss: -14.22757115653867\n",
      "  node_ip: 127.0.0.1\n",
      "  pid: 46526\n",
      "  time_since_restore: 11.351194143295288\n",
      "  time_this_iter_s: 0.10791015625\n",
      "  time_total_s: 11.351194143295288\n",
      "  timestamp: 1658499901\n",
      "  timesteps_since_restore: 0\n",
      "  training_iteration: 100\n",
      "  trial_id: 09e21122\n",
      "  warmup_time: 0.003228902816772461\n",
      "  \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Result for objective_12309db2:\n",
      "  date: 2022-07-22_15-25-03\n",
      "  done: false\n",
      "  experiment_id: 3915d19b775c4ee1843b5dcf79560a93\n",
      "  hostname: Kais-MacBook-Pro.local\n",
      "  iterations: 0\n",
      "  iterations_since_restore: 1\n",
      "  mean_loss: 10.503337773185708\n",
      "  neg_mean_loss: -10.503337773185708\n",
      "  node_ip: 127.0.0.1\n",
      "  pid: 46549\n",
      "  time_since_restore: 0.10407328605651855\n",
      "  time_this_iter_s: 0.10407328605651855\n",
      "  time_total_s: 0.10407328605651855\n",
      "  timestamp: 1658499903\n",
      "  timesteps_since_restore: 0\n",
      "  training_iteration: 1\n",
      "  trial_id: 12309db2\n",
      "  warmup_time: 0.0030128955841064453\n",
      "  \n",
      "Result for objective_12342770:\n",
      "  date: 2022-07-22_15-25-03\n",
      "  done: false\n",
      "  experiment_id: 1da16f0c20d7438b93927280dc40e5a1\n",
      "  hostname: Kais-MacBook-Pro.local\n",
      "  iterations: 0\n",
      "  iterations_since_restore: 1\n",
      "  mean_loss: 1.1922491879354844\n",
      "  neg_mean_loss: -1.1922491879354844\n",
      "  node_ip: 127.0.0.1\n",
      "  pid: 46550\n",
      "  time_since_restore: 0.1050269603729248\n",
      "  time_this_iter_s: 0.1050269603729248\n",
      "  time_total_s: 0.1050269603729248\n",
      "  timestamp: 1658499903\n",
      "  timesteps_since_restore: 0\n",
      "  training_iteration: 1\n",
      "  trial_id: '12342770'\n",
      "  warmup_time: 0.0032460689544677734\n",
      "  \n",
      "Result for objective_12374d7e:\n",
      "  date: 2022-07-22_15-25-03\n",
      "  done: false\n",
      "  experiment_id: 5788d010ee194eeeabfc3592d37fb2cc\n",
      "  hostname: Kais-MacBook-Pro.local\n",
      "  iterations: 0\n",
      "  iterations_since_restore: 1\n",
      "  mean_loss: 19.839240921278268\n",
      "  neg_mean_loss: -19.839240921278268\n",
      "  node_ip: 127.0.0.1\n",
      "  pid: 46551\n",
      "  time_since_restore: 0.10313534736633301\n",
      "  time_this_iter_s: 0.10313534736633301\n",
      "  time_total_s: 0.10313534736633301\n",
      "  timestamp: 1658499903\n",
      "  timesteps_since_restore: 0\n",
      "  training_iteration: 1\n",
      "  trial_id: 12374d7e\n",
      "  warmup_time: 0.002891063690185547\n",
      "  \n",
      "Result for objective_1045958e:\n",
      "  date: 2022-07-22_15-25-05\n",
      "  done: false\n",
      "  experiment_id: 3579bfc2b346424b833f82f23d459807\n",
      "  hostname: Kais-MacBook-Pro.local\n",
      "  iterations: 47\n",
      "  iterations_since_restore: 48\n",
      "  mean_loss: 3.0263684682219036\n",
      "  neg_mean_loss: -3.0263684682219036\n",
      "  node_ip: 127.0.0.1\n",
      "  pid: 46544\n",
      "  time_since_restore: 5.180821180343628\n",
      "  time_this_iter_s: 0.10701394081115723\n",
      "  time_total_s: 5.180821180343628\n",
      "  timestamp: 1658499905\n",
      "  timesteps_since_restore: 0\n",
      "  training_iteration: 48\n",
      "  trial_id: 1045958e\n",
      "  warmup_time: 0.002847909927368164\n",
      "  \n",
      "Result for objective_12309db2:\n",
      "  date: 2022-07-22_15-25-08\n",
      "  done: false\n",
      "  experiment_id: 3915d19b775c4ee1843b5dcf79560a93\n",
      "  hostname: Kais-MacBook-Pro.local\n",
      "  iterations: 47\n",
      "  iterations_since_restore: 48\n",
      "  mean_loss: 0.6271166903395393\n",
      "  neg_mean_loss: -0.6271166903395393\n",
      "  node_ip: 127.0.0.1\n",
      "  pid: 46549\n",
      "  time_since_restore: 5.172047138214111\n",
      "  time_this_iter_s: 0.11187624931335449\n",
      "  time_total_s: 5.172047138214111\n",
      "  timestamp: 1658499908\n",
      "  timesteps_since_restore: 0\n",
      "  training_iteration: 48\n",
      "  trial_id: 12309db2\n",
      "  warmup_time: 0.0030128955841064453\n",
      "  \n",
      "Result for objective_12342770:\n",
      "  date: 2022-07-22_15-25-08\n",
      "  done: false\n",
      "  experiment_id: 1da16f0c20d7438b93927280dc40e5a1\n",
      "  hostname: Kais-MacBook-Pro.local\n",
      "  iterations: 47\n",
      "  iterations_since_restore: 48\n",
      "  mean_loss: -8.685737519988487\n",
      "  neg_mean_loss: 8.685737519988487\n",
      "  node_ip: 127.0.0.1\n",
      "  pid: 46550\n",
      "  time_since_restore: 5.172597885131836\n",
      "  time_this_iter_s: 0.10711097717285156\n",
      "  time_total_s: 5.172597885131836\n",
      "  timestamp: 1658499908\n",
      "  timesteps_since_restore: 0\n",
      "  training_iteration: 48\n",
      "  trial_id: '12342770'\n",
      "  warmup_time: 0.0032460689544677734\n",
      "  \n",
      "Result for objective_12374d7e:\n",
      "  date: 2022-07-22_15-25-08\n",
      "  done: false\n",
      "  experiment_id: 5788d010ee194eeeabfc3592d37fb2cc\n",
      "  hostname: Kais-MacBook-Pro.local\n",
      "  iterations: 47\n",
      "  iterations_since_restore: 48\n",
      "  mean_loss: 9.955526689542863\n",
      "  neg_mean_loss: -9.955526689542863\n",
      "  node_ip: 127.0.0.1\n",
      "  pid: 46551\n",
      "  time_since_restore: 5.162422180175781\n",
      "  time_this_iter_s: 0.10872411727905273\n",
      "  time_total_s: 5.162422180175781\n",
      "  timestamp: 1658499908\n",
      "  timesteps_since_restore: 0\n",
      "  training_iteration: 48\n",
      "  trial_id: 12374d7e\n",
      "  warmup_time: 0.002891063690185547\n",
      "  \n",
      "Result for objective_1045958e:\n",
      "  date: 2022-07-22_15-25-10\n",
      "  done: false\n",
      "  experiment_id: 3579bfc2b346424b833f82f23d459807\n",
      "  hostname: Kais-MacBook-Pro.local\n",
      "  iterations: 94\n",
      "  iterations_since_restore: 95\n",
      "  mean_loss: 2.852294770731789\n",
      "  neg_mean_loss: -2.852294770731789\n",
      "  node_ip: 127.0.0.1\n",
      "  pid: 46544\n",
      "  time_since_restore: 10.196197271347046\n",
      "  time_this_iter_s: 0.10780715942382812\n",
      "  time_total_s: 10.196197271347046\n",
      "  timestamp: 1658499910\n",
      "  timesteps_since_restore: 0\n",
      "  training_iteration: 95\n",
      "  trial_id: 1045958e\n",
      "  warmup_time: 0.002847909927368164\n",
      "  \n",
      "Result for objective_1045958e:\n",
      "  date: 2022-07-22_15-25-11\n",
      "  done: true\n",
      "  experiment_id: 3579bfc2b346424b833f82f23d459807\n",
      "  experiment_tag: 5_activation=relu,height=-73.2818,steps=100,width=5.7883\n",
      "  hostname: Kais-MacBook-Pro.local\n",
      "  iterations: 99\n",
      "  iterations_since_restore: 100\n",
      "  mean_loss: 2.8433363404373386\n",
      "  neg_mean_loss: -2.8433363404373386\n",
      "  node_ip: 127.0.0.1\n",
      "  pid: 46544\n",
      "  time_since_restore: 10.737220287322998\n",
      "  time_this_iter_s: 0.10835909843444824\n",
      "  time_total_s: 10.737220287322998\n",
      "  timestamp: 1658499911\n",
      "  timesteps_since_restore: 0\n",
      "  training_iteration: 100\n",
      "  trial_id: 1045958e\n",
      "  warmup_time: 0.002847909927368164\n",
      "  \n",
      "Result for objective_18344524:\n",
      "  date: 2022-07-22_15-25-13\n",
      "  done: false\n",
      "  experiment_id: 10740ef080664baaa8cca7ba5cb899ac\n",
      "  hostname: Kais-MacBook-Pro.local\n",
      "  iterations: 0\n",
      "  iterations_since_restore: 1\n",
      "  mean_loss: 6.887162536491742\n",
      "  neg_mean_loss: -6.887162536491742\n",
      "  node_ip: 127.0.0.1\n",
      "  pid: 46569\n",
      "  time_since_restore: 0.10450887680053711\n",
      "  time_this_iter_s: 0.10450887680053711\n",
      "  time_total_s: 0.10450887680053711\n",
      "  timestamp: 1658499913\n",
      "  timesteps_since_restore: 0\n",
      "  training_iteration: 1\n",
      "  trial_id: '18344524'\n",
      "  warmup_time: 0.002858877182006836\n",
      "  \n",
      "Result for objective_12309db2:\n",
      "  date: 2022-07-22_15-25-13\n",
      "  done: false\n",
      "  experiment_id: 3915d19b775c4ee1843b5dcf79560a93\n",
      "  hostname: Kais-MacBook-Pro.local\n",
      "  iterations: 94\n",
      "  iterations_since_restore: 95\n",
      "  mean_loss: 0.5656126475885976\n",
      "  neg_mean_loss: -0.5656126475885976\n",
      "  node_ip: 127.0.0.1\n",
      "  pid: 46549\n",
      "  time_since_restore: 10.193128108978271\n",
      "  time_this_iter_s: 0.10602211952209473\n",
      "  time_total_s: 10.193128108978271\n",
      "  timestamp: 1658499913\n",
      "  timesteps_since_restore: 0\n",
      "  training_iteration: 95\n",
      "  trial_id: 12309db2\n",
      "  warmup_time: 0.0030128955841064453\n",
      "  \n",
      "Result for objective_12342770:\n",
      "  date: 2022-07-22_15-25-13\n",
      "  done: false\n",
      "  experiment_id: 1da16f0c20d7438b93927280dc40e5a1\n",
      "  hostname: Kais-MacBook-Pro.local\n",
      "  iterations: 94\n",
      "  iterations_since_restore: 95\n",
      "  mean_loss: -8.746369700451542\n",
      "  neg_mean_loss: 8.746369700451542\n",
      "  node_ip: 127.0.0.1\n",
      "  pid: 46550\n",
      "  time_since_restore: 10.210996866226196\n",
      "  time_this_iter_s: 0.10719585418701172\n",
      "  time_total_s: 10.210996866226196\n",
      "  timestamp: 1658499913\n",
      "  timesteps_since_restore: 0\n",
      "  training_iteration: 95\n",
      "  trial_id: '12342770'\n",
      "  warmup_time: 0.0032460689544677734\n",
      "  \n",
      "Result for objective_12374d7e:\n",
      "  date: 2022-07-22_15-25-13\n",
      "  done: false\n",
      "  experiment_id: 5788d010ee194eeeabfc3592d37fb2cc\n",
      "  hostname: Kais-MacBook-Pro.local\n",
      "  iterations: 94\n",
      "  iterations_since_restore: 95\n",
      "  mean_loss: 9.897723841978765\n",
      "  neg_mean_loss: -9.897723841978765\n",
      "  node_ip: 127.0.0.1\n",
      "  pid: 46551\n",
      "  time_since_restore: 10.19912338256836\n",
      "  time_this_iter_s: 0.10725522041320801\n",
      "  time_total_s: 10.19912338256836\n",
      "  timestamp: 1658499913\n",
      "  timesteps_since_restore: 0\n",
      "  training_iteration: 95\n",
      "  trial_id: 12374d7e\n",
      "  warmup_time: 0.002891063690185547\n",
      "  \n",
      "Result for objective_12309db2:\n",
      "  date: 2022-07-22_15-25-14\n",
      "  done: true\n",
      "  experiment_id: 3915d19b775c4ee1843b5dcf79560a93\n",
      "  experiment_tag: 6_activation=relu,height=-94.9666,steps=100,width=16.9764\n",
      "  hostname: Kais-MacBook-Pro.local\n",
      "  iterations: 99\n",
      "  iterations_since_restore: 100\n",
      "  mean_loss: 0.5624860552037738\n",
      "  neg_mean_loss: -0.5624860552037738\n",
      "  node_ip: 127.0.0.1\n",
      "  pid: 46549\n",
      "  time_since_restore: 10.73293399810791\n",
      "  time_this_iter_s: 0.1079857349395752\n",
      "  time_total_s: 10.73293399810791\n",
      "  timestamp: 1658499914\n",
      "  timesteps_since_restore: 0\n",
      "  training_iteration: 100\n",
      "  trial_id: 12309db2\n",
      "  warmup_time: 0.0030128955841064453\n",
      "  \n",
      "Result for objective_12342770:\n",
      "  date: 2022-07-22_15-25-14\n",
      "  done: true\n",
      "  experiment_id: 1da16f0c20d7438b93927280dc40e5a1\n",
      "  experiment_tag: 7_activation=tanh,height=-98.0775,steps=100,width=17.2252\n",
      "  hostname: Kais-MacBook-Pro.local\n",
      "  iterations: 99\n",
      "  iterations_since_restore: 100\n",
      "  mean_loss: -8.749451683536464\n",
      "  neg_mean_loss: 8.749451683536464\n",
      "  node_ip: 127.0.0.1\n",
      "  pid: 46550\n",
      "  time_since_restore: 10.745534181594849\n",
      "  time_this_iter_s: 0.10716795921325684\n",
      "  time_total_s: 10.745534181594849\n",
      "  timestamp: 1658499914\n",
      "  timesteps_since_restore: 0\n",
      "  training_iteration: 100\n",
      "  trial_id: '12342770'\n",
      "  warmup_time: 0.0032460689544677734\n",
      "  \n",
      "Result for objective_12374d7e:\n",
      "  date: 2022-07-22_15-25-14\n",
      "  done: true\n",
      "  experiment_id: 5788d010ee194eeeabfc3592d37fb2cc\n",
      "  experiment_tag: 8_activation=relu,height=-1.6076,steps=100,width=18.0841\n",
      "  hostname: Kais-MacBook-Pro.local\n",
      "  iterations: 99\n",
      "  iterations_since_restore: 100\n",
      "  mean_loss: 9.894786565536863\n",
      "  neg_mean_loss: -9.894786565536863\n",
      "  node_ip: 127.0.0.1\n",
      "  pid: 46551\n",
      "  time_since_restore: 10.734816074371338\n",
      "  time_this_iter_s: 0.10638594627380371\n",
      "  time_total_s: 10.734816074371338\n",
      "  timestamp: 1658499914\n",
      "  timesteps_since_restore: 0\n",
      "  training_iteration: 100\n",
      "  trial_id: 12374d7e\n",
      "  warmup_time: 0.002891063690185547\n",
      "  \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Result for objective_1a1e29b8:\n",
      "  date: 2022-07-22_15-25-17\n",
      "  done: false\n",
      "  experiment_id: 26fc8c5f05a6407690dc8cada561576e\n",
      "  hostname: Kais-MacBook-Pro.local\n",
      "  iterations: 0\n",
      "  iterations_since_restore: 1\n",
      "  mean_loss: 17.40289288319415\n",
      "  neg_mean_loss: -17.40289288319415\n",
      "  node_ip: 127.0.0.1\n",
      "  pid: 46576\n",
      "  time_since_restore: 0.10475707054138184\n",
      "  time_this_iter_s: 0.10475707054138184\n",
      "  time_total_s: 0.10475707054138184\n",
      "  timestamp: 1658499917\n",
      "  timesteps_since_restore: 0\n",
      "  training_iteration: 1\n",
      "  trial_id: 1a1e29b8\n",
      "  warmup_time: 0.0028810501098632812\n",
      "  \n",
      "Result for objective_18344524:\n",
      "  date: 2022-07-22_15-25-19\n",
      "  done: false\n",
      "  experiment_id: 10740ef080664baaa8cca7ba5cb899ac\n",
      "  hostname: Kais-MacBook-Pro.local\n",
      "  iterations: 30\n",
      "  iterations_since_restore: 31\n",
      "  mean_loss: -2.8488858516331668\n",
      "  neg_mean_loss: 2.8488858516331668\n",
      "  node_ip: 127.0.0.1\n",
      "  pid: 46569\n",
      "  time_since_restore: 5.212157964706421\n",
      "  time_this_iter_s: 0.10646200180053711\n",
      "  time_total_s: 5.212157964706421\n",
      "  timestamp: 1658499919\n",
      "  timesteps_since_restore: 0\n",
      "  training_iteration: 31\n",
      "  trial_id: '18344524'\n",
      "  warmup_time: 0.002858877182006836\n",
      "  \n",
      "Result for objective_1a1e29b8:\n",
      "  date: 2022-07-22_15-25-22\n",
      "  done: false\n",
      "  experiment_id: 26fc8c5f05a6407690dc8cada561576e\n",
      "  hostname: Kais-MacBook-Pro.local\n",
      "  iterations: 47\n",
      "  iterations_since_restore: 48\n",
      "  mean_loss: 7.610248309424407\n",
      "  neg_mean_loss: -7.610248309424407\n",
      "  node_ip: 127.0.0.1\n",
      "  pid: 46576\n",
      "  time_since_restore: 5.170850038528442\n",
      "  time_this_iter_s: 0.1072230339050293\n",
      "  time_total_s: 5.170850038528442\n",
      "  timestamp: 1658499922\n",
      "  timesteps_since_restore: 0\n",
      "  training_iteration: 48\n",
      "  trial_id: 1a1e29b8\n",
      "  warmup_time: 0.0028810501098632812\n",
      "  \n",
      "Result for objective_18344524:\n",
      "  date: 2022-07-22_15-25-24\n",
      "  done: false\n",
      "  experiment_id: 10740ef080664baaa8cca7ba5cb899ac\n",
      "  hostname: Kais-MacBook-Pro.local\n",
      "  iterations: 77\n",
      "  iterations_since_restore: 78\n",
      "  mean_loss: -3.0083151799393004\n",
      "  neg_mean_loss: 3.0083151799393004\n",
      "  node_ip: 127.0.0.1\n",
      "  pid: 46569\n",
      "  time_since_restore: 10.263540983200073\n",
      "  time_this_iter_s: 0.10780215263366699\n",
      "  time_total_s: 10.263540983200073\n",
      "  timestamp: 1658499924\n",
      "  timesteps_since_restore: 0\n",
      "  training_iteration: 78\n",
      "  trial_id: '18344524'\n",
      "  warmup_time: 0.002858877182006836\n",
      "  \n",
      "Result for objective_18344524:\n",
      "  date: 2022-07-22_15-25-26\n",
      "  done: true\n",
      "  experiment_id: 10740ef080664baaa8cca7ba5cb899ac\n",
      "  experiment_tag: 9_activation=tanh,height=-41.1284,steps=100,width=12.2952\n",
      "  hostname: Kais-MacBook-Pro.local\n",
      "  iterations: 99\n",
      "  iterations_since_restore: 100\n",
      "  mean_loss: -3.0313530888899516\n",
      "  neg_mean_loss: 3.0313530888899516\n",
      "  node_ip: 127.0.0.1\n",
      "  pid: 46569\n",
      "  time_since_restore: 12.62198805809021\n",
      "  time_this_iter_s: 0.1080331802368164\n",
      "  time_total_s: 12.62198805809021\n",
      "  timestamp: 1658499926\n",
      "  timesteps_since_restore: 0\n",
      "  training_iteration: 100\n",
      "  trial_id: '18344524'\n",
      "  warmup_time: 0.002858877182006836\n",
      "  \n",
      "Result for objective_1a1e29b8:\n",
      "  date: 2022-07-22_15-25-27\n",
      "  done: false\n",
      "  experiment_id: 26fc8c5f05a6407690dc8cada561576e\n",
      "  hostname: Kais-MacBook-Pro.local\n",
      "  iterations: 93\n",
      "  iterations_since_restore: 94\n",
      "  mean_loss: 7.5087713304782575\n",
      "  neg_mean_loss: -7.5087713304782575\n",
      "  node_ip: 127.0.0.1\n",
      "  pid: 46576\n",
      "  time_since_restore: 10.177540063858032\n",
      "  time_this_iter_s: 0.1076350212097168\n",
      "  time_total_s: 10.177540063858032\n",
      "  timestamp: 1658499927\n",
      "  timesteps_since_restore: 0\n",
      "  training_iteration: 94\n",
      "  trial_id: 1a1e29b8\n",
      "  warmup_time: 0.0028810501098632812\n",
      "  \n",
      "Result for objective_1a1e29b8:\n",
      "  date: 2022-07-22_15-25-27\n",
      "  done: true\n",
      "  experiment_id: 26fc8c5f05a6407690dc8cada561576e\n",
      "  experiment_tag: 10_activation=tanh,height=64.0289,steps=100,width=10.0482\n",
      "  hostname: Kais-MacBook-Pro.local\n",
      "  iterations: 99\n",
      "  iterations_since_restore: 100\n",
      "  mean_loss: 7.502418319119348\n",
      "  neg_mean_loss: -7.502418319119348\n",
      "  node_ip: 127.0.0.1\n",
      "  pid: 46576\n",
      "  time_since_restore: 10.823745012283325\n",
      "  time_this_iter_s: 0.10851097106933594\n",
      "  time_total_s: 10.823745012283325\n",
      "  timestamp: 1658499927\n",
      "  timesteps_since_restore: 0\n",
      "  training_iteration: 100\n",
      "  trial_id: 1a1e29b8\n",
      "  warmup_time: 0.0028810501098632812\n",
      "  \n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:ray.tune.tune:Total run time: 43.33 seconds (43.21 seconds for the tuning loop).\n"
     ]
    }
   ],
   "source": [
    "tuner = tune.Tuner(\n",
    "    objective,\n",
    "    tune_config=tune.TuneConfig(\n",
    "#         metric=\"mean_loss\",\n",
    "#         mode=\"min\",\n",
    "        search_alg=algo,\n",
    "        num_samples=num_samples,\n",
    "    ),\n",
    "    param_space={\"steps\": 100},\n",
    ")\n",
    "results = tuner.fit()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "64a17648",
   "metadata": {},
   "source": [
    "Here are the hyperparameters found to minimize the mean loss of the defined objective. Note that we have to pass the metric and mode here because we don't set it in the TuneConfig."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "aac3e88b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best hyperparameters found were:  {'steps': 100, 'width': 17.225166732233465, 'height': -98.07750812064515, 'activation': 'tanh'}\n"
     ]
    }
   ],
   "source": [
    "print(\"Best hyperparameters found were: \", results.get_best_result(\"mean_loss\", \"min\").config)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "0478c1ea",
   "metadata": {
    "tags": [
     "remove-cell"
    ]
   },
   "outputs": [],
   "source": [
    "ray.shutdown()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  },
  "orphan": true
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
